Skip to content

DIAGNOSTICS ONLY — DO NOT MERGE: trace Linux outbound attach release - #30

Closed
myobie wants to merge 2 commits into
mainfrom
diag/linux-outbound-attach-release
Closed

DIAGNOSTICS ONLY — DO NOT MERGE: trace Linux outbound attach release#30
myobie wants to merge 2 commits into
mainfrom
diag/linux-outbound-attach-release

Conversation

@myobie

@myobie myobie commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Diagnostics-only branch to run the exact Linux CI job and locate where the outbound attach teardown stalls on Linux.

Do not merge. No production behaviour change; instrumentation and one control test only.

Context: attached_outbound_session_blocks_recycle_and_a_dead_one_does_not fails on Linux only, at its negative control. Measured locally on macOS the gauge releases in ~53ms (3/3 runs); Linux CI did not release within 20s — ~380x, so not runner slowness.

This branch traces the chain (local EOF → client send-close → remote PTY child exit → server close → client observes remote-close → attach return → gauge decrement) using the client gauge and the server's tunnel session stats, over a widened 40s window so a slow-but-working release is distinguishable from a permanent stall. It also adds a bounded control proving the instrumentation does not itself release the gauge.

myobie added 2 commits August 3, 2026 11:42
Instruments the one test that fails on Linux and only Linux, to locate which
link of the teardown chain stalls: local EOF, client send-close, remote PTY child
exit, server close, client observing remote-close, attach return, gauge decrement.

The client gauge is the observable for the end of that chain and the server's
tunnel session stats are the observable for its middle, so a stall in the server
detaching is distinguishable from a stall in the client's attach returning.

The observation window is widened from 20s to 40s on purpose, so a slow but
working release is distinguishable from a permanent stall.

Also adds a bounded control proving the instrumentation does not itself release
the gauge: it holds a live outbound session, runs the same polling shape, and
asserts the gauge stays up throughout and the session still answers afterwards.
Without that, the trace could be measuring its own side effects.

No production behaviour change. macOS locally: releases at 251ms with the server
showing detached=1, and the control holds at gauge=1 across 3s of polling.
The previous Linux run PASSED this test with the widened 40s window, which is
itself the headline finding: the gauge does release on Linux, so 20s was too
tight rather than the release being broken. But cargo only shows a test's
captured output when it fails, so passing hid the actual timing.

This panics with the measured numbers so they reach the log either way. Local
macOS baseline: released=true elapsed_ms=290, server detached=1.
@myobie

myobie commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

Closing: this did its job.

What it proved. Forced tracing on Linux CI showed the outbound tunnel teardown was a product defect, not a test oracle defect: released=false at elapsed_ms=40001 with srv_active_attaches=1 and srv_detached=0, so the server still counted the session attached 40s after the client dropped its local socket. macOS detached in ~53ms. That measurement is what ruled out the oracle hypothesis and turned issue #32 into a real bug with a known blast radius.

What fixed it. b6ffc0a (PR #38) — the abrupt-close branch now calls mark_send_closed, so it records the final offset and the writer emits Frame::Close. Root cause was a missing message, not a slow path: a dropped socket reads as clean EOF on macOS and an error on Linux, and only the EOF branch reported the close.

The permanent guard is tunnel::tests::an_abrupt_local_close_reports_the_end_just_like_a_clean_eof, which injects the EOF-vs-error decision directly and so fails on every platform when the defect is present — no CI timing involved.

Superseded, not abandoned. Deleting the branch now.

@myobie myobie closed this Aug 4, 2026
@myobie
myobie deleted the diag/linux-outbound-attach-release branch August 4, 2026 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant